gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
5943 commits 28 branches 278 tags 135 MiB
Commit graph

739 commits

Author SHA1 Message Date
Chris Lu 678c54d705 data sink: add incremental mode 2021-02-28 16:19:03 -08:00
Chris Lu 62191b08ea disk type support custom tags 2021-02-22 02:03:12 -08:00
bingoohuang 50df484d86 support command line arguments in a configuration file so we can weed master -config=master.conf, weed volume -config=volume.conf and etc. 2021-02-20 16:52:57 +08:00
Chris Lu e1992c83de webdav adds disktype 2021-02-18 15:45:44 -08:00
Chris Lu c2ad6f1047 webdav add replication setting 
fix https://github.com/chrislusf/seaweedfs/issues/1817
 2021-02-18 12:15:09 -08:00
Chris Lu 3575d41009 go fmt 2021-02-17 20:57:08 -08:00
Chris Lu 6daa932f5c refactoring to get master function, instead of passing master values directly 
this will enable retrying later
 2021-02-17 20:55:55 -08:00
Chris Lu f8446b42ab this can compile now!!! 2021-02-16 02:47:02 -08:00
LazyDBA247-Anyvision 7f458d5e78 better postgres connection pool management 
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime)
to enable refresh of stale connections.
 2021-02-15 07:45:09 +02:00
Chris Lu 0bc3a1f9e8 disk type only supports hdd and ssd, not ready for random tags yet 2021-02-14 11:38:43 -08:00
LazyDBA247-Anyvision 51b4963e2e postgres2 & memsql2 
add escape (quote identifiers) for the dynamic sql
so tables (collections) with special characters will work.
 2021-02-14 13:14:36 +02:00
Chris Lu ef76365ec2 adjust help message 2021-02-13 15:47:08 -08:00
Chris Lu 4bd8a692d8 disk type can be generic tags 2021-02-13 13:50:14 -08:00
Chris Lu 7d9dc3c6a2 use fasthttp lib to read 2021-02-12 05:59:52 -08:00
Chris Lu 821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
bingoohuang 7256902fb0 fix typo offset.ToAcutalOffset to offset.ToActualOffset 2021-02-07 12:11:51 +08:00
Chris Lu d67ccb66c0 webdav: can start together with "weed server" or "weed filer" 2021-01-31 22:16:52 -08:00
Chris Lu d475c89fcc go fmt 2021-01-28 15:23:46 -08:00
Chris Lu 19295600f9 mount: change option name to volumeServerAccess, with publicUrl and filerProxy modes 2021-01-28 15:23:16 -08:00
Chris Lu 9a06c35da4 replicate: incremental sink only contains new and updated files 
address da08402ba2
 2021-01-28 02:39:22 -08:00
Chris Lu da08402ba2 replicate: use creation time for local incremental file sink 
related to https://github.com/chrislusf/seaweedfs/pull/1762
 2021-01-28 02:17:41 -08:00
Chris Lu 822f1ade9d
Merge pull request #1762 from kmlebedev/backupsink 
replication to create time date directory
 2021-01-28 02:03:33 -08:00
Konstantin Lebedev be1062b7fc rename in scaffold 2021-01-28 14:59:20 +05:00
Konstantin Lebedev 02fdc0a333 rename backup to local_incremental and use mtime 2021-01-28 14:56:13 +05:00
Chris Lu 5f72c388bc
Merge pull request #1764 from danlsgiga/handle-umask 
Use provided umask on mount
 2021-01-28 01:09:20 -08:00
Daniel Santos d38613c49a match upstream 2021-01-27 20:56:41 -07:00
Konstantin Lebedev 6b54ff9912 replication to create time date directory 2021-01-27 15:01:33 +05:00
Chris Lu 0488c84bea adjust help message 2021-01-27 00:17:52 -08:00
Konstantin Lebedev 3634811408 replication to todays date directory 2021-01-27 12:45:58 +05:00
Chris Lu 5e07afb0f0
Merge pull request #1759 from kmlebedev/sink.local 
replication to local disk storage
 2021-01-26 22:31:54 -08:00
Daniel Santos 0c64d60fdb Fix log message with correct mode 2021-01-26 21:45:52 -07:00
Daniel Santos 0aaaa9b470 Fix var assignment 2021-01-26 21:06:08 -07:00
Daniel Santos 5a761bb7a4 Enforce umask correctly 2021-01-26 20:42:10 -07:00
Chris Lu ad2a20c8a5 notification add ack and nack 2021-01-26 11:08:44 -08:00
Konstantin Lebedev 612b7975a1 replication to local disk storage 2021-01-26 22:50:25 +05:00
Chris Lu 3a1d3d3413 mount: properly invalidate kernel node cache entry 
fix https://github.com/chrislusf/seaweedfs/issues/1752
 2021-01-26 02:50:53 -08:00
Chris Lu cc1f3907ff filer.sync: default filerProxy to true 2021-01-25 10:14:42 -08:00
Chris Lu 62ebb917e0 mount: avoid the need to specify different cache directory for different mount 
fix https://github.com/seaweedfs/seaweedfs-csi-driver/issues/11
 2021-01-25 09:10:20 -08:00
Chris Lu 00707ec00f mount: outsideContainerClusterMode proxy through filer 
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
 2021-01-24 19:01:58 -08:00
Chris Lu 6ca10725b8 Revert "mount: when outside cluster network, use filer as proxy to access volume servers" 
This reverts commit 096e088d7b.
 2021-01-24 03:15:19 -08:00
Chris Lu 096e088d7b mount: when outside cluster network, use filer as proxy to access volume servers 2021-01-24 01:41:38 -08:00
Chris Lu 80b8692688 filer.sync: replicate outside of either cluster, only need to see filers 2021-01-24 00:01:44 -08:00
Chris Lu 024c186a48 filer.meta.tail: output json format for easier parsing 2021-01-21 22:39:25 -08:00
Chris Lu 84f05787f8 mount: limit background requests and congestion threshold 2021-01-20 23:30:14 -08:00
Chris Lu 1a3f859c58 filer store: postgres configuration adjust text 2021-01-19 18:12:07 -08:00
Chris Lu b434f7e4e0 filer store: postgres adjust default config, add optioanl schema 2021-01-19 18:10:36 -08:00
Chris Lu d5add83e85 filer store: add postgres2 2021-01-19 18:07:29 -08:00
Chris Lu 52a8f1470e filer store: add mysql2 2021-01-19 17:21:50 -08:00
Chris Lu 96354208c5 scaffold: add YugabyteDB 2021-01-19 12:35:14 -08:00
Chris Lu f13c082583 add memsql to comments 2021-01-19 11:31:55 -08:00
LazyDBA247-Anyvision 8eed763b97 better mysql connection pool management 
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime)
to enable refresh of connections.
 2021-01-14 08:14:21 +02:00
Chris Lu 3fb2ed9093 filer.meta.tail: optionally submit metadata changes to ElasticSearch 2021-01-13 03:31:37 -08:00
Chris Lu c5df2577f5 rename 2021-01-13 00:31:19 -08:00
Chris Lu 90c5077610 rename "weed watch" to "weed filer.meta.tail" 2021-01-12 18:48:01 -08:00
Chris Lu 0ef43a23a7 go fmt 2021-01-12 02:30:12 -08:00
Chris Lu 629c9962e7 filer: add leveldb3 
support dedicated leveldb instance for each bucket
 2021-01-12 02:29:44 -08:00
Chris Lu cfb9342a15 avoid concurrent map updates to viper 2021-01-12 02:28:13 -08:00
Chris Lu 2c7148cd55 redis cluster: do not enable route by latency by default 2021-01-11 02:43:53 -08:00
Chris Lu 6e12a3a490 refactoring 2021-01-11 02:08:26 -08:00
Chris Lu 62ee484d12 refactoring 2021-01-11 00:03:13 -08:00
Chris Lu 1efb51ba84 filer: change to saveToFilerLimit from cacheToFilerLimit 
short circuit saving small files to volume server
 2021-01-10 23:14:46 -08:00
Chris Lu d3d3f2fb9b mount: default to 128 concurrent writers 2021-01-09 22:53:37 -08:00
Chris Lu 30efaa66c6 server: add option to disable master 2021-01-07 02:35:47 -08:00
Chris Lu 2b76854641 add "weed filer.cat" to read files directly from volume servers 2021-01-06 04:22:00 -08:00
Chris Lu 5f538c5f9d Merge branch 'add_rocksdb' 2021-01-03 00:48:56 -08:00
Chris Lu a04d8927a8 add scaffold 2021-01-03 00:44:52 -08:00
Chris Lu 4250a3a675 upload: add read default collection from master 
fix https://github.com/chrislusf/seaweedfs/issues/1724#issuecomment-753576071
 2021-01-02 23:43:01 -08:00
Chris Lu adef4ddc87 mount: load security keys 
fix https://github.com/chrislusf/seaweedfs/issues/1714
 2020-12-29 11:51:38 -08:00
Chris Lu 89977123b6 shell: add shell.toml support 
fix https://github.com/chrislusf/seaweedfs/issues/1664
 2020-12-28 15:07:18 -08:00
Chris Lu 90df52d001 s3: add command option "allowEmptyFolder" to filer and server 2020-12-27 21:09:06 -08:00
Chris Lu 561a2ff0bc s3: add option for "alllowEmptyFolder" 2020-12-27 10:50:27 -08:00
Chris Lu 75c6edba9e filer: hbase add enabled flag 2020-12-24 00:19:16 -08:00
Chris Lu c3d1b3b5aa hook up 2020-12-23 21:49:01 -08:00
Chris Lu 1c7e1295dc Merge branch 'master' into support_ssd_volume 2020-12-22 17:44:52 -08:00
Chris Lu a1009e8044 filer: add -defaultStoreDir so that filer.toml can be skipped 
fix https://github.com/chrislusf/seaweedfs/issues/1659
 2020-12-22 17:33:40 -08:00
Chris Lu 90d785a15f filer: redis, redis cluster, cassandra support super large directory 2020-12-22 02:26:05 -08:00
Chris Lu 41c0f3ad24 filer: support path-specific filer store 2020-12-19 01:27:09 -08:00
Chris Lu 1bf22c0b5b go fmt 2020-12-16 09:14:05 -08:00
Chris Lu 23014b6810 Merge branch 'master' into support_ssd_volume 2020-12-16 08:26:51 -08:00
Chris Lu 3c7f9633eb show error a little bit more obviously 
fix https://github.com/chrislusf/seaweedfs/issues/1678
 2020-12-14 09:39:33 -08:00
Chris Lu f6a419c26c disk type configurable for each folder 2020-12-13 22:49:56 -08:00
Chris Lu 7c52a35942 refactor 2020-12-13 22:29:52 -08:00
Chris Lu 8baba93fce rename parameter name to "disk" 2020-12-13 12:06:21 -08:00
Chris Lu 51eadaf2b6 rename parameter name to "disk" 2020-12-13 12:05:31 -08:00
Chris Lu 0d2ec832e2 rename from volumeType to diskType 2020-12-13 11:59:32 -08:00
Chris Lu d156c74ec0 volume server set volume type and heartbeat to the master 2020-12-13 03:11:24 -08:00
Chris Lu e9cd798bd3 adding volume type 2020-12-13 00:58:58 -08:00
Chris Lu 06bb7bf6c0 adding /usr/local/etc/seaweedfs for bsd style config search path 
related to https://reviews.freebsd.org/D27391
 2020-12-07 16:46:48 -08:00
Chris Lu 263eb29e9f filer: add option to cache small files to filer store 2020-12-04 22:39:43 -08:00
Chris Lu 6d30b21b10 volume: add "-dir.idx" option for separate index storage 
fix https://github.com/chrislusf/seaweedfs/issues/1265
 2020-11-27 03:17:10 -08:00
Chris Lu 983fd90010 marked as WIP 2020-11-26 15:08:52 -08:00
Chris Lu b11449f955 pass in dir.idx parameter to DiskLocation 2020-11-26 14:59:03 -08:00
Chris Lu ef3e068c51 deprecating "filer.options.buckets_fsync" in filer.toml 2020-11-16 16:56:49 -08:00
Chris Lu 827930e249
Merge pull request #1610 from kmlebedev/prefer_read_in_this_dc 
prefer to read from volumes in this data center
 2020-11-11 22:09:41 -08:00
Konstantin Lebedev 0880aff224 update dc flag desc 2020-11-12 10:07:52 +05:00
Konstantin Lebedev fc7baef5bb fiil serverUrls sorted by data center 2020-11-12 02:13:33 +05:00
Chris Lu 745ee8d8f3 avoid unnecessary error message 
fix https://github.com/chrislusf/seaweedfs/issues/1611
 2020-11-11 13:00:05 -08:00
Chris Lu 8750cac090 move to util.RetryWaitTime 2020-11-01 02:36:43 -08:00
Chris Lu de86945aeb go fmt 2020-10-31 16:45:38 -07:00
Chris Lu 8826601be1 mount: optional limit for the number of concurrent writers 2020-10-30 21:22:20 -07:00
Chris Lu 7ecbb4b3c9 clean up 2020-10-30 10:46:31 -07:00
Kenny 9cc6575ab5
Update filer_copy.go 
修改 weed filer.copy 上传目录时文件名变子目录的问题
 2020-10-30 18:16:49 +08:00
Chris Lu 47047516d6 Revert "increase default volume file size limit to 1024" 
This reverts commit 0983060a

I am confused why this was changed to 1024 by myself. 1GB should be too large for most cases.
 2020-10-29 15:46:29 -07:00
Chris Lu c6d9974fe7 server: Add option to enable/disable volume server 
fix https://github.com/chrislusf/seaweedfs/issues/1572
 2020-10-29 00:24:18 -07:00
Chris Lu e219c57849 passing full path when assign volume locations 2020-10-25 15:46:29 -07:00
Chris Lu f375b93aef renaming 2020-10-25 15:32:43 -07:00
Chris Lu 63b0fb54f7 minor 2020-10-25 14:15:53 -07:00
Chris Lu 147d4d23b6 filer.copy: automatically use the right bucket 2020-10-25 11:21:33 -07:00
Chris Lu da36abf033 go fmt 2020-10-24 20:12:04 -07:00
Chris Lu b7b1eb51b1 default to 2 for better performance in common computer hardware 2020-10-24 20:08:26 -07:00
Chris Lu 1b17f71939 adjust election timeout to 10 seconds 2020-10-23 23:06:44 -07:00
Chris Lu 0cea84e7ab make it easy with setting up multiple masters in a clcuster 2020-10-23 18:18:46 -07:00
Konstantin Lebedev a9a7005687 set desc of option 2020-10-22 11:23:00 +05:00
Chris Lu 6c92fa349d filer: support default rack 
fix https://github.com/chrislusf/seaweedfs/issues/1546
 2020-10-20 17:41:39 -07:00
Chris Lu ee1fc6558a refactor 2020-10-16 11:02:44 -07:00
Chris Lu 3f7d1d1bf1 Only wait on retryable requests 2020-10-13 00:29:46 -07:00
Chris Lu 7704469d29 mount: outsideContainerClusterMode changed to use volume server publicUrl 2020-10-11 20:42:15 -07:00
Chris Lu d155f907c2 mount: configurable read wait time 2020-10-10 20:09:43 -07:00
Chris Lu 78859e35fc s3: adjust config examples 2020-10-08 10:12:09 -07:00
Chris Lu a8624c2e4f read from alternative replica 
related to https://github.com/chrislusf/seaweedfs/issues/1512
 2020-10-07 22:49:04 -07:00
Chris Lu da4edf3651 master: check peers for existing leader before starting a leader election 
fix https://github.com/chrislusf/seaweedfs/issues/1509
 2020-10-07 01:25:39 -07:00
Chris Lu c543762e23 volume: auto expands -max setting 
I have tried to run weed volume but got errors...
cmd:
weed volume -max=0 -mserver="127.0.0.1:9333" -port=8080 -dir="/var/www/d1/sfs,/var/www/d2/sfs,/var/www/d3/sfs"
log:
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d1/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d2/sfs Permission: -rwxr-xr-x
I1006 15:09:26 17040 file_util.go:23] Folder /var/www/d3/sfs Permission: -rwxr-xr-x
F1006 15:09:26 17040 volume.go:142] 3 directories by -dir, but only 1 max is set by -max
What am I doing wrong? Thx.
 2020-10-06 09:05:30 -07:00
Chris Lu 2c20ef72ae fix starting problem 2020-10-03 13:56:38 -07:00
Chris Lu d77e3c96e9 FUSE: use the mount directory permissions 2020-10-03 13:37:35 -07:00
Устюжанин Антон Александрович 6fbfe73683 fix: resumeState 2020-10-03 22:16:47 +05:00
Устюжанин Антон Александрович dc31b19469 fix: restore raft state 2020-10-03 14:03:41 +05:00
Устюжанин Антон Александрович 8c82fb7e5f fix: restore raft state 2020-10-02 23:01:20 +05:00
Chris Lu 79ab10e300 adjust help message 2020-09-30 09:32:00 -07:00
Chris Lu 55cb68c09c filer: replication follows master config if not specified 2020-09-30 09:15:55 -07:00
Chris Lu e6552b5e1e filer: able to start s3 together 2020-09-27 23:00:43 -07:00
Chris Lu 8516517c48 filer store: Cassandra supports username/password 2020-09-24 21:31:06 -07:00
Chris Lu 043b063136
Merge pull request #1482 from hilimd/master 
Fix: s3 delete object
 2020-09-24 18:21:34 -07:00
limd 48c578410f mount: rollback default value 2020-09-25 09:18:52 +08:00
Chris Lu 4856bce0ee adjust for metrics port 2020-09-24 10:21:23 -07:00
Chris Lu d2d3aec3e1 consolidate to one metricsPort in "weed server" mode 2020-09-24 09:55:02 -07:00
Chris Lu 2e9099369e
Merge pull request #1489 from kmlebedev/promhttp 
Promhttp
 2020-09-24 09:50:48 -07:00
Konstantin Lebedev 98e9de6e11 fix style 2020-09-24 17:48:39 +05:00
Konstantin Lebedev 324e44d4b3 add start metrics server 2020-09-24 17:45:39 +05:00
limd cbca14edc5 mount: fix k8s pvc and os mount directory permission bug 2020-09-24 18:07:16 +08:00
limd 59e91e9c7e mount: fix k8s pvc mount directory permission 2020-09-24 10:25:45 +08:00
limd 9b21ec27e4 mount: auto created dir set corrct umask fix bug 2020-09-23 18:35:37 +08:00
Chris Lu f7a0ccb595 mount: auto created dir set corrct umask 2020-09-23 02:31:19 -07:00
James Hartig 658fc2e5b6 Allow option to enable volume pprof on server 2020-09-21 22:43:10 -04:00
Chris Lu d013d09a9b adjust logging 2020-09-20 15:38:59 -07:00
Chris Lu 2c21eb1971 volume: get metrics configuration from master 
fix https://github.com/chrislusf/seaweedfs/issues/1354
 2020-09-19 00:03:00 -07:00
Chris Lu 6544e60bea s3 add metrics 
empty for now
 2020-09-17 06:56:15 -07:00
Chris Lu feca07bf96 Filer: ip bind 
fix https://github.com/chrislusf/seaweedfs/issues/1470
 2020-09-16 10:26:08 -07:00
Chris Lu 684a875876 update help message 2020-09-15 00:40:38 -07:00
Chris Lu 103fafe00b weed export: print out [start,stop) content range 2020-09-14 22:57:23 -07:00
Chris Lu 5d6753fb98 shell: add volumeServer.leave command 2020-09-13 21:25:51 -07:00